In [1]:
__nbid__ = '0042'
__author__ = 'Brian Merino <brian.merino@noirlab.edu>, Vinicius Placco <vinicius.placco@noirlab.edu>'
__version__ = '20240606' # yyyymmdd; version datestamp of this notebook
__keywords__ = ['gmos','gemini','longslit','whitedwarf','dragons']

Gemini GMOS longslit white dwarf reduction using DRAGONS Python API¶

Public archival data from GS-2017B-Q-72 (J2145+0031)¶

adapted from https://dragons.readthedocs.io/projects/gmosls-drtutorial/en/v3.1.0/ex1_gmosls_dithered_api.html¶

Table of contents¶

  • Goals
  • Summary
  • Disclaimers and attribution
  • Imports and setup
  • About the dataset
  • Downloading data for reduction
  • Set up the DRAGONS logger
  • Create File Lists
  • Create Bad Pixel Mask
  • Create Master Bias
  • Create Master Flat Field
  • Processed Arc
  • Processed Standard
  • Science Observations
  • Display the 2D Spectrum
  • Display 1-D flux-calibrated spectrum
  • Clean-up (optional)

Goals¶

Showcase how to reduce GMOS longslit data using the Gemini DRAGONS package on the Data Lab science platform using a custom DRAGONS kernel "DRAGONS (Py3.7)". The steps include downloading data from the Gemini archive, setting up a DRAGONS calibration service, processing flats, darks, a bad pixel mask, and science frames, and creating a single combined stacked image.

Summary¶

DRAGONS is a Python-based astronomical data reduction platform written by the Gemini Science User Support Department. It can currently be used to reduce imaging data from Gemini instruments GMOS, NIRI, Flamingos 2, GSAOI, and GNIRS, as well as spectroscopic data in GMOS longslit mode. Linked here is a general list of guides, manuals, and tutorials about the use of DRAGONS.

The DRAGONS kernel has been made available in the Data Lab environment, allowing users to access the routines without being dependent on installing the software on their local machines.

In this notebook, we present an example of a DRAGONS Jupyter notebook that works in the Data Lab environment to reduce example Gemini South GMOS longslit data fully. This notebook will not present all of the details of the many options available to adjust or optimize the DRAGONS GMOS longslit data reduction process; rather, it will just show one example of a standard reduction of a GMOS longslit dataset.

The data used in this notebook example is GMOS longslit data from the Gemini archive of the candiate DB white dwarf J2145+0031 from the Gemini South program "Spectroscopy of massive DB white dwarf candidate stars", PI: Alejandra Romero, program ID GS-2017B-Q-72.

Disclaimer & attribution¶

Disclaimers¶

Note that using the Astro Data Lab constitutes your agreement with our minimal Disclaimers.

Acknowledgments¶

If you use Astro Data Lab in your published research, please include the text in your paper's Acknowledgments section:

This research uses services or data provided by the Astro Data Lab, which is part of the Community Science and Data Center (CSDC) Program of NSF NOIRLab. NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA), Inc. under a cooperative agreement with the U.S. National Science Foundation.

If you use SPARCL jointly with the Astro Data Lab platform (via JupyterLab, command-line, or web interface) in your published research, please include this text below in your paper's Acknowledgments section:

This research uses services or data provided by the SPectra Analysis and Retrievable Catalog Lab (SPARCL) and the Astro Data Lab, which are both part of the Community Science and Data Center (CSDC) Program of NSF NOIRLab. NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA), Inc. under a cooperative agreement with the U.S. National Science Foundation.

In either case please cite the following papers:

  • Data Lab concept paper: Fitzpatrick et al., "The NOAO Data Laboratory: a conceptual overview", SPIE, 9149, 2014, https://doi.org/10.1117/12.2057445

  • Astro Data Lab overview: Nikutta et al., "Data Lab - A Community Science Platform", Astronomy and Computing, 33, 2020, https://doi.org/10.1016/j.ascom.2020.100411

If you are referring to the Data Lab JupyterLab / Jupyter Notebooks, cite:

  • Juneau et al., "Jupyter-Enabled Astrophysical Analysis Using Data-Proximate Computing Platforms", CiSE, 23, 15, 2021, https://doi.org/10.1109/MCSE.2021.3057097

If publishing in a AAS journal, also add the keyword: \facility{Astro Data Lab}

And if you are using SPARCL, please also add \software{SPARCL} and cite:

  • Juneau et al., "SPARCL: SPectra Analysis and Retrievable Catalog Lab", Conference Proceedings for ADASS XXXIII, 2024 https://doi.org/10.48550/arXiv.2401.05576

The NOIRLab Library maintains lists of proper acknowledgments to use when publishing papers using the Lab's facilities, data, or services.

For this notebook specifically, please acknowledge:

  • DRAGONS publication: Labrie et al., "DRAGONS - Data Reduction for Astronomy from Gemini Observatory North and South", ASPC, 523, 321L

  • DRAGONS open source software publication

Importing Python libraries¶

In [2]:
import warnings
import glob

import astrodata

from astropy.io import fits
from astropy.utils.exceptions import AstropyWarning

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import LogNorm

import gemini_instruments
from gempy.utils import logutils
from gempy.adlibrary import dataselect
from gempy.adlibrary import plotting

from recipe_system import cal_service
from recipe_system.reduction.coreReduce import Reduce

warnings.simplefilter('ignore', category=AstropyWarning)
warnings.filterwarnings("ignore")

About the dataset¶

The table below contains a summary of the dataset:

Observation Type File name(s) Wavelength
Science S20171022S0087-89
S20171022S0095-97		 515 nm
530 nm
Science biases S20171021S0265-269
S20171023S0032-036
Science flats S20171022S0088 (515 nm)
S20171022S0096 (530 nm)		 515 nm
530 nm
Science arcs S20171022S0092
S20171022S0099		 515 nm
530 nm
Standard (LTT2415) S20170826S0160 515 nm
Standard biases S20170825S0347-351
S20170826S0224-228
Standard flats S20170826S0161 515 nm
Standard arc S20170826S0162 515 nm
BPM bpm_20140601_gmos-s_Ham_22_full_12amp.fits

Downloading the data¶

Downloading longslit data from the Gemini archive to the current working directory. This step only needs to be executed once.

If you run this notebook for the first time and need to download the dataset, set the variable "download=True". The notebook will not redownload the dataset if it is set to False. This will become particularly useful if you run the notebooks more than once.

In [3]:
%%bash 

# create file that lists FITS files to be downloaded
echo "\
http://archive.gemini.edu/file/S20171022S0087.fits
http://archive.gemini.edu/file/S20171022S0088.fits
http://archive.gemini.edu/file/S20171022S0089.fits
http://archive.gemini.edu/file/S20171022S0095.fits
http://archive.gemini.edu/file/S20171022S0096.fits
http://archive.gemini.edu/file/S20171022S0097.fits
http://archive.gemini.edu/file/S20171021S0265.fits
http://archive.gemini.edu/file/S20171021S0266.fits
http://archive.gemini.edu/file/S20171021S0267.fits
http://archive.gemini.edu/file/S20171021S0268.fits
http://archive.gemini.edu/file/S20171021S0269.fits
http://archive.gemini.edu/file/S20171023S0032.fits
http://archive.gemini.edu/file/S20171023S0033.fits
http://archive.gemini.edu/file/S20171023S0034.fits
http://archive.gemini.edu/file/S20171023S0035.fits
http://archive.gemini.edu/file/S20171023S0036.fits
http://archive.gemini.edu/file/S20171022S0088.fits
http://archive.gemini.edu/file/S20171022S0096.fits
http://archive.gemini.edu/file/S20171022S0092.fits
http://archive.gemini.edu/file/S20171022S0099.fits
http://archive.gemini.edu/file/S20170826S0160.fits
http://archive.gemini.edu/file/S20170825S0347.fits
http://archive.gemini.edu/file/S20170825S0348.fits
http://archive.gemini.edu/file/S20170825S0349.fits
http://archive.gemini.edu/file/S20170825S0350.fits
http://archive.gemini.edu/file/S20170825S0351.fits
http://archive.gemini.edu/file/S20170826S0224.fits
http://archive.gemini.edu/file/S20170826S0225.fits
http://archive.gemini.edu/file/S20170826S0226.fits
http://archive.gemini.edu/file/S20170826S0227.fits
http://archive.gemini.edu/file/S20170826S0228.fits
http://archive.gemini.edu/file/S20170826S0161.fits
http://archive.gemini.edu/file/S20170826S0162.fits
http://archive.gemini.edu/file/bpm_20140601_gmos-s_Ham_22_full_12amp.fits\
" > gmos_ls.list
In [4]:
%%bash

download="True"

if [ $download == "True" ]; then
    wget --no-check-certificate -N -q -i gmos_ls.list

else
    echo "Skipping download. To download the data set used in this notebook, set download=True."
fi

Create a list of all the FITS files in the directory

In [5]:
all_files = glob.glob('S2017*.fits')
all_files.append(glob.glob('bpm*.fits')[0])
all_files.sort()

Setting up the DRAGONS logger¶

DRAGONS comes with a local calibration manager that uses the same calibration association rules as the Gemini Observatory Archive. This allows reduce to make requests to a local light-weight database for matching processed calibrations when needed to reduce a dataset.

This tells the system where to put the calibration database. This database will keep track of the processed calibrations we will send to it.

In [6]:
logutils.config(file_name='gmosls.log')
caldb = cal_service.set_local_database()
caldb.init("w")

Create file lists¶

This data set contains science and calibration frames. For some programs, it could have different observed targets and exposure times depending on how you organize your raw data.

The DRAGONS data reduction pipeline does not organize the data for you. You have to do it. DRAGONS provides tools to help you with that.

The first step is to create lists that will be used in the data reduction process. For that, we use dataselect. Please refer to the dataselect documentation for details regarding its usage.

In [7]:
all_biases = dataselect.select_data(all_files, ['BIAS'])
for bias in all_biases:
    ad = astrodata.open(bias)
    print(bias, '  ', ad.detector_roi_setting())

S20170825S0347.fits    Central Spectrum
S20170825S0348.fits    Central Spectrum
S20170825S0349.fits    Central Spectrum
S20170825S0350.fits    Central Spectrum
S20170825S0351.fits    Central Spectrum
S20170826S0224.fits    Central Spectrum
S20170826S0225.fits    Central Spectrum
S20170826S0226.fits    Central Spectrum
S20170826S0227.fits    Central Spectrum
S20170826S0228.fits    Central Spectrum
S20171021S0265.fits    Full Frame
S20171021S0266.fits    Full Frame
S20171021S0267.fits    Full Frame
S20171021S0268.fits    Full Frame
S20171021S0269.fits    Full Frame
S20171023S0032.fits    Full Frame
S20171023S0033.fits    Full Frame
S20171023S0034.fits    Full Frame
S20171023S0035.fits    Full Frame
S20171023S0036.fits    Full Frame
In [8]:
biasstd = dataselect.select_data(
    all_files,
    ['BIAS'],
    [],
    dataselect.expr_parser('detector_roi_setting=="Central Spectrum"')
)

biassci = dataselect.select_data(
    all_files,
    ['BIAS'],
    [],
    dataselect.expr_parser('detector_roi_setting=="Full Frame"')
)

A list for the flats¶

The GMOS longslit flats are not normally stacked. The default recipe does not stack the flats. This allows us to use only one list of the flats. Each will be reduced individually, never interacting with the others.

In [9]:
flats = dataselect.select_data(all_files, ['FLAT'])

A list for the arcs¶

The GMOS longslit arcs are not normally stacked. The default recipe does not stack the arcs. This allows us to use only one list of arcs. Each will be reduce individually, never interacting with the others.

In [10]:
arcs = dataselect.select_data(all_files, ['ARC'])

A list for the spectrophotometric standard star¶

If a spectrophotometric standard is recognized as such by DRAGONS, it will receive the Astrodata tag STANDARD. To be recognized, the name of the star must be in a lookup table. All spectrophotometric standards normally used at Gemini are in that table.

In [11]:
stdstar = dataselect.select_data(all_files, ['STANDARD'])

A list for the science observation¶

The science observations are what is left, that is anything that is not a calibration. Calibrations are assigned the astrodata tag CAL, therefore we can select against that tag to get the science observations.

In [12]:
scitarget = dataselect.select_data(
    all_files,
    [],
    ['CAL'],
    dataselect.expr_parser('object=="J2145+0031"')
)

Bad pixel mask¶

Starting with DRAGONS v3.1, the static bad pixel masks (BPMs) are now handled as calibrations. They are downloadable from the archive instead of being packaged with the software. They are automatically associated like any other calibrations. This means that the user now must download the BPMs along with the other calibrations and add the BPMs to the local calibration manager.

In [13]:
for bpm in dataselect.select_data(all_files, ['BPM']):
    caldb.add_cal(bpm)

Create master bias¶

We create the master biases with the Reduce class. We will run it twice, once for each of the two raw bias lists. The master biases will be automatically added to the local calibration manager when the "store" parameter is present in the .dragonsrc configuration file. The output is written to disk and its name is stored in the Reduce instance. The calibration service expects the name of a file on disk.

Because the database was given the "store" option in the dragonsrc file, the processed biases will be automatically added to the database at the end of the recipe.

When the cell is done running, the master biases will have the name of the first bias on each list, both with the suffix _bias.fits.

In [14]:
reduce_biasstd = Reduce()
reduce_biassci = Reduce()
reduce_biasstd.files.extend(biasstd)
reduce_biassci.files.extend(biassci)
reduce_biasstd.runr()
reduce_biassci.runr()

All submitted files appear valid:
S20170825S0347.fits ... S20170826S0228.fits, 10 files submitted.

Ginga not installed, use other viewer, or no viewer
No description has been provided for this image No description has been provided for this image

Create master flat field¶

GMOS longslit flat fields are normally obtained at night along with the observation sequence to match the telescope and instrument flexure. The matching flat nearest in time to the target observation is used to flat field the target. The central wavelength, filter, grating, binning, gain, and read speed must match.

Because of the flexure, GMOS longslit flat fields are not stacked. Each is reduced and used individually. The default recipe takes that into account.

We can send all the flats, regardless of characteristics, to Reduce and each will be reduce individually. When a calibration is needed, in this case, a master bias, the best match will be obtained automatically from the local calibration manager.

The master flats will have the name of the first flat of each list, all with the suffix _flat.fits

NOTE: A warning may appear after running this cell regarding too few unmasked points. This warning can be ignored and will not affect the final product.

In [15]:
reduce_flats = Reduce()
reduce_flats.files.extend(flats)
reduce_flats.runr()

   WARNING - Too few unmasked points. Unmasking all data.

Processed arc - wavelength solution¶

GMOS longslit arc can be obtained at night with the observation sequence, if requested by the program, but are often obtained at the end of the night or the following afternoon instead. In this example, the arcs have been obtained at night, as part of the sequence. Like the spectroscopic flats, they are not stacked which means that they can be sent to reduce all together and will be reduced individually.

The wavelength solution is automatically calculated and the algorithm has been found to be quite reliable. There might be cases where it fails; inspect the *_mosaic.pdf plot and the RMS of determineWavelengthSolution in the logs to confirm a good solution.

The processed arcs will be saved with the suffix _arc.fits. The mosaics will be saved with the suffix _mosaic.pdf.

In [16]:
reduce_arcs = Reduce()
reduce_arcs.files.extend(arcs)
reduce_arcs.runr()

Processed standard - sensitivity function¶

The GMOS longslit spectrophotometric standards are normally taken when there is a hole in the queue schedule, often when the weather is not good enough for science observations. One standard per configuration, per program is the norm. If you dither along the dispersion axis, most likely only one of the positions will have been used for the spectrophotometric standard. This is normal for baseline calibrations at Gemini. The standard is used to calculate the sensitivity function. It has been shown that a difference of 10 or so nanometers does not significantly impact the spectrophotometric calibration.

The reduction of the standard will be using a BPM, a master bias, a master flat, and a processed arc. If those have been added to the local calibration manager, they will be picked up automatically. The output of the reduction includes the sensitivity function and will be added to the calibration database automatically if the "store" option is set in the dragonsrc configuration file.

The processed standard will be saved with the suffix _standard.fits.

In [17]:
reduce_std = Reduce()
reduce_std.files.extend(stdstar)
reduce_std.runr()

Science observations¶

The science target is a DB white dwarf candidate. The sequence has four images that were dithered spatially and along the dispersion axis. DRAGONS will register the four images in both directions, align and stack them before extracting the 1-D spectrum.

With the master bias, the master flat, the processed arcs (one for each of the grating position, aka central wavelength), and the processed standard in the local calibration manager, one only needs to do as follows to reduce the science observations and extract the 1-D spectrum.

Running this cell will produce the final reduced 2D spectrum with the suffix _2D.fits, as well as several files that contain the names of the science images with the suffix _flagCosmicRays.pdf.

Warning: This cell may take approximately 10 minutes to finish running.

In [18]:
reduce_science = Reduce()
reduce_science.files.extend(scitarget)
reduce_science.runr()

Display the 2D spectrum¶

In [19]:
file = 'S20171022S0087_2D.fits'

hdu = fits.open(file)
spectrum = hdu[1].data

# You can choose the scaling of the 2D spectrum by uncommenting the first line below
# for linear sclaing, or the line below that for logarithmic scaling. 
plt.imshow(spectrum,origin='lower',cmap='Greys_r',vmin=-5,vmax=15) #Linear
#plt.imshow(spectrum,origin='lower',cmap='Greys_r',norm=LogNorm(vmin=0.005,vmax=1500)) #Logarithmic

plt.xlim(500,3250)
plt.ylim(100,2000)

plt.xlabel('Detector Position [pixels]',fontweight='bold',fontsize=14)
plt.ylabel('Detector Position [pixels]',fontweight='bold',fontsize=14)
plt.title('%s'%file,fontweight='bold',fontsize=16)
plt.show()
No description has been provided for this image

ASCII representation¶

If you need an ASCII representation of the spectrum, you can use the primitive write1DSpectra to extract the values from the FITS file.

NOTE: Running this cell may result in two harmless warnings that can be ignored without affecting the quality of the spectrum.

In [20]:
writeascii = Reduce()
writeascii.files = ['S20171022S0087_1D.fits']
writeascii.recipename = 'write1DSpectra'
writeascii.runr()

WARNING - No recipe can be found in gmos recipe libs.
WARNING - Searching primitives ...

Display the 1-D flux-calibrated spectrum of our sole target¶

In [21]:
ad = astrodata.open('S20171022S0087_1D.fits')

data = ad[0].data
wavelength = ad[0].wcs(np.arange(data.size)).astype(np.float32)
units = ad[0].wcs.output_frame.unit[0]

plt.xlabel(f'Wavelength ({units})')
plt.ylabel(f'Signal ({ad[0].hdr["BUNIT"]})')
plt.ylim(0,10**(-17))
plt.plot(wavelength, data)
plt.show()
No description has been provided for this image

Optional: remove duplicate calibrations and remove raw data (uncomment lines before running)¶

In [23]:
# %%bash

# cp S20171022S0087_2D.fits final.fits
# cp S20171022S0087_1D.fits final_1D.fits
# rm -r calibrations
# rm *dat
# rm gmos_ls.list gmosls.log S2017*
# rm bpm_20140601_gmos-s_Ham_22_full_12amp.fits
# mv final.fits S20171022S0087_2D.fits
# mv final_1D.fits S20171022S0087_1D.fits
In [ ]: